crawler_to_md/scraper.py (1)

47-50: LGTM! Proxy configuration implemented correctly.

The implementation properly:

• Uses requests.Session for better connection management
• Configures proxy for both HTTP and HTTPS protocols
• Stores the proxy reference for potential future use

Consider adding basic URL validation for the proxy parameter to catch common formatting errors early.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

tests/test_cli.py (4)

51-85: Refactor to reduce code duplication.

The fake_init function and monkeypatching setup are duplicated across multiple test functions. Consider extracting common setup logic into a helper function or fixture.

+def _setup_proxy_test(monkeypatch, captured):
+    def fake_init(
+        self,
+        base_url,
+        exclude_patterns,
+        db_manager,
+        rate_limit=0,
+        delay=0,
+        proxy=None,
+    ):
+        captured['proxy'] = proxy
+
+    monkeypatch.setattr(Scraper, '__init__', fake_init)
+    monkeypatch.setattr(Scraper, 'start_scraping', lambda *a, **k: None)
+    monkeypatch.setattr(ExportManager, 'export_to_markdown', lambda *a, **k: None)
+    monkeypatch.setattr(ExportManager, 'export_to_json', lambda *a, **k: None)
+
 def test_cli_proxy_option(monkeypatch, tmp_path):
     captured = {}
-
-    def fake_init(
-        self,
-        base_url,
-        exclude_patterns,
-        db_manager,
-        rate_limit=0,
-        delay=0,
-        proxy=None,
-    ):
-        captured['proxy'] = proxy
-
-    monkeypatch.setattr(Scraper, '__init__', fake_init)
-    monkeypatch.setattr(Scraper, 'start_scraping', lambda *a, **k: None)
-    monkeypatch.setattr(ExportManager, 'export_to_markdown', lambda *a, **k: None)
-    monkeypatch.setattr(ExportManager, 'export_to_json', lambda *a, **k: None)
+    _setup_proxy_test(monkeypatch, captured)

---

87-121: Use consistent quote style.

The test uses single quotes while the rest of the file uses double quotes. Maintain consistency with the existing codebase.

-    cache_folder = tmp_path / 'cache'
+    cache_folder = tmp_path / "cache"
-        'prog',
-        '--url',
-        'http://example.com',
-        '--output-folder',
+        "prog",
+        "--url",
+        "http://example.com",
+        "--output-folder",

---

159-179: Consider adding more specific error validation.

The test only verifies that SystemExit is raised but doesn't validate the error message or exit code. Consider adding assertions for more specific error handling.

+import contextlib
+import io
+
 def test_cli_proxy_error(monkeypatch, tmp_path):
     def fake_init(*a, **k):
         raise ValueError('Proxy unreachable')
 
     monkeypatch.setattr(Scraper, '__init__', fake_init)
+    
+    # Capture stderr to verify error message
+    stderr_capture = io.StringIO()
+    
     cache_folder = tmp_path / 'cache'
     args = [
         'prog',
         '--url',
         'http://example.com',
         '--output-folder',
         str(tmp_path),
         '--cache-folder',
         str(cache_folder),
         '--proxy',
         'http://proxy:8080',
     ]
     monkeypatch.setattr(sys, 'argv', args)
-    with pytest.raises(SystemExit):
-        cli.main()
+    
+    with contextlib.redirect_stderr(stderr_capture):
+        with pytest.raises(SystemExit) as exc_info:
+            cli.main()
+    
+    assert exc_info.value.code == 1
+    assert 'Proxy unreachable' in stderr_capture.getvalue()

---

51-179: Consider adding edge case tests.

The current test suite covers the main scenarios but could benefit from additional edge cases to ensure robustness.

Additional test cases to consider:

• Empty proxy string
• Malformed proxy URLs
• Proxy with authentication
• IPv6 proxy addresses

def test_cli_proxy_empty_string(monkeypatch, tmp_path):
    # Test behavior with empty proxy string
    pass

def test_cli_proxy_malformed_url(monkeypatch, tmp_path):
    # Test behavior with malformed proxy URL
    pass

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro